In-conference question summary system

ABSTRACT

A communication system adapted to allow participants, during a collaboration event or session, to ask questions while also serving up or providing relevant context to the presenter. This can be done by the participant clicking anywhere on the screen of their display on their client device during an online presentation and then presenting their question (e.g., typing or otherwise entering their input) about that particular portion of the presentation. When a user clicks on their screen during the presentation, a new dialog will open with a screenshot of the point of the presentation that occurred as well as a dialog box to type in a question. When the participant submits their question, a question report can be created and sent to the server. This report may include a user identifier (ID), a timestamp of when the screenshot was taken, the coordinates of the click/selection location, and the question itself.

FIELD OF THE INVENTION

The present disclosure generally relates to electronic communicationmethods and systems including those utilizing the Internet andcloud-based solutions. More particularly, examples of the disclosurerelate to electronic communication methods and systems that are adaptedto facilitate the efficient presentation and answering of questionssubmitted during an presentation or event (e.g., a video conferencecall, a training session, a facilitated meeting, and the like) that isprovided over a digital communications network (e.g., via the Internetas a cloud-based service).

BACKGROUND OF THE DISCLOSURE

Mobile and web applications allow users, such as operators ofsmartphones and other portable computing devices as well as tablets,laptop, notebook, and desktop computers, to perform a wide range ofcommunication functions. In recent years, there has been a rapidexpansion of the use mobile and web applications to allow two-to-manypeople in varying locations to communicate with each other in aconcurrent manner or as a group.

For example, video conferencing and similar applications have rapidlyexpanded in popularity, and these applications allow the users to seeeach other and share information. These applications are being used forbusiness and other meetings, for remote learning classes for primaryschools, secondary schools, universities, and other organizations, formarketing presentations, for courtroom sessions, and for many othersettings where one or more facilitators or presenters wish to present orshare both visual and audio content and to interact with theparticipants (or application “users”). Such video conferencingapplications often allow screen sharing and messaging to occur duringthe video conference or web-based meeting.

While providing many advantages for disseminating information, thereremain unresolved issues with using video conferencing or similarapplications. During a presentation to a group of people, some of theaudience members or participants may have questions they wish topresent. For example, they may wish to get further details about aparticular portion of the presentation (e.g., not understand how tosolve a problem in an online math class, have not heard the presenter'spoint during a business meeting, and so on) or may wish to provide inputto the presenter (e.g., the “question” may be a counterpoint or an addedstatement that agrees with the content being presented at a particulartime during a meeting or class). The portion related to the question maybe presented by a slide, a live demo, a shared video clip, or simply bya speaker (such as the presenter/facilitator/teacher or another audiencemember/user) talking. Many video conferencing applications do notadequately support users (or audience members) presenting questions(which may include comments or nearly any input as the term is usedherein).

In some presentations, the presenter(s) may provide a question andanswer (Q&A) session at the end of the presentation. In this case,though, the audience members may forget their questions or the portionfor which the question pertains as they must wait until the end of thepresentation to ask their question (e.g., have a question about contentpresented in first few minutes of an hour long class or meeting). From apresenter's perspective, they may lose track of the context in which thequestion was framed or be confused as to which portion of thepresentation the question pertains.

In other presentation formats, the users or audience members are allowedto interrupt (e.g., press a button on their client device to “raisetheir hands” or indicate they have a question or simply begin talking toask their question) the presenter during the presentation, but thisoften is undesirable as it can disrupt the presentation and distract thepresenter. In some cases, the users can submit questions during thepresentation, but these “live” questions are often ignored or notaddressed by the presenter in real time or are saved by the videoconferencing application for review at the end of the presentation.Again, such a delayed Q&A approach can result in the context being lostmaking it more difficult for the presenter to provide an adequate orappropriate response.

Any discussion of problems provided in this section has been included inthis disclosure solely for the purposes of providing a background forthe present invention and should not be taken as an admission that anyor all of the discussion was known at the time the invention was made.

BRIEF DESCRIPTION OF THE DRAWING FIGURES

The subject matter of the present disclosure is particularly pointed outand distinctly claimed in the concluding portion of the specification. Amore complete understanding of the present disclosure, however, may bestbe obtained by referring to the detailed description and claims whenconsidered in connection with the drawing figures, wherein like numeralsdenote like elements and wherein:

FIG. 1 illustrates an electronic, cloud-based communication systemadapted to provide enhanced collaborative communications includingin-conference (or in-session) question summaries in accordance withexemplary embodiments of the disclosure.

FIGS. 2A and 2B illustrate a display device operating to displaycollaboration session content (a slide during a presentation or thelike) and to display a dialog to facilitate a participant entering aquestion.

FIG. 3 is a flow diagram of a method of in-conference questionsummarization as may be performed by a communication or collaborationsystem of the present description such as the system shown in FIG. 1.

FIG. 4 is another flow diagram of a method of in-conference questionsummarization as may be performed by a communication or collaborationsystem of the present description similar to that shown in FIG. 3 butshowing the user of NLP subprocessing.

FIG. 5 is a flow diagram showing an exemplary implementation of the NLPsubprocessing that may be used in the methods of FIGS. 3 and 4 and asmay be carried out by the operation of the system of FIG. 1.

It will be appreciated that elements in the figures are illustrated forsimplicity and clarity and have not necessarily been drawn to scale. Forexample, the dimensions of some of the elements in the figures may beexaggerated relative to other elements to help to improve understandingof illustrated embodiments of the present invention.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

The description of exemplary embodiments of the present inventionprovided below is merely exemplary and is intended for purposes ofillustration only; the following description is not intended to limitthe scope of the invention disclosed herein. Moreover, recitation ofmultiple embodiments having stated features is not intended to excludeother embodiments having additional features or other embodimentsincorporating different combinations of the stated features.

As set forth in more detail below, exemplary embodiments of thedisclosure relate to electronic communication systems, and correspondingmethods performed by such systems, that can, for example, provideenhanced features for online communication conferences or sessions. Forexample, the new system (and corresponding methods) are particularlywell suited for allowing participants of a video conferencing session toprovide input, such as questions, to the presenter in an improved mannerthat is less disruptive to the presenter of the session and to itsparticipants while retaining the context of the user input so as toallow more efficient and accurate responses to be generated during orafter the video conferencing session. Hence, the new system may bethought of as “in-conference question summary system.”

The new system (and associated method) allows participants, during acollaboration event or session, to ask questions (or provide other userinput) while also serving up or providing relevant context to thepresenter. This can be done by the participant clicking anywhere on thescreen of their display on their client device during an onlinepresentation and then presenting their question (e.g., typing orotherwise entering their input via their client device's input/outputcomponent) about that particular portion of the presentation. When auser clicks on their screen (or otherwise interacts with their display)during the presentation, a new dialog will open with a screenshot of thepoint of the presentation that occurred as well as a dialog box to typein a question.

The exact location of the screen that the user clicked upon orselected/indicated may be determined by the system and identified to theuser in this dialog screenshot. The coordinates of the screen/displaywhere they clicked will also be recorded and stored as part of thequestion/user input report. This is helpful in case the presenter movesto the next slide or other portion of the presentation as the pertinentportion in the presentation will still be visible to theuser/participant in the separate dialog. When they submit theirquestion, a question report can be created and sent to the server. Thisreport may include a user identifier (ID), a timestamp of when thescreenshot was taken, the coordinates of the click/selection location,and the question itself.

Once the server receives the question report, it may respond byassociating it against the current meeting/presentation. The server maythen act to examine the timestamp of other questions that have beensubmitted. If they are determined by the server to be close in time(e.g., within 30 seconds to 5 minutes or longer to suit a type ofpresentation and its content (or presenter's selection of a timewindow)), the server may then be configured to look at the coordinatesof the screen location and, if they are close in proximity to anotherinput question, it will look at the question, e.g., using a naturallanguage processing (NLP) tool or the like, to determine if thequestions are related (e.g., determine if keywords are the same orsimilar based on the analysis). If determine to be related questions,the server may present only one of the questions (e.g., the one that isfirst or last in time) or combine the two (or more) questions into asingle question to avoid duplication (while, in some cases, indicatingthat multiple participants who each may be identified to the presenterhave asked this question or had this similar input).

From the organizer's or presenter's application (e.g., a collaborationutility or application (“app”) running on their client device), theywill have the ability to view the question report list at any time. Thequestion report list may be served to their device by the server and beautomatically updated with any question reports received and processedduring the duration of the presentation as well as afterward during amore conventional Q&A portion of the presentation. In some defaultoperating modes, the question report list is ordered by the screenshottimestamp. Then, when the presenter is viewing a question report, thepresenter can see the screenshot which includes when the user clicked(e.g., time the question was raised by the participant) as well as thequestion/user input itself and who submitted it (the participant's ID).The presenter can then answer all or a subset of the questions knowingthe context (timing and presentation content related to the question) ofeach question.

With this overview in mind, FIG. 1 illustrates an electronic,cloud-based communication system 100 adapted to provide enhancedcollaborative communications including in-conference (or in-session)question summaries in accordance with exemplary embodiments of thedisclosure. As shown, the system 100 includes a web server 110communicatively linked via its application programming interface (API)116 and a digital communications network 105 in a wired or wirelessmanner shown with arrows 131, 151 with one-to-many participant meeting(or client) devices 130 and one or more presenter meeting (or client)devices 150.

The server 110 may be any useful server device (or devices as thefunctionality may be provided with two or more such devices) as wellknown in the arts for hosting or facilitating sharing of content betweenthe presenter's device 150 and the participant's devices 130. The clientdevices 130 and 150 can be or can include any suitable device with wiredor wireless communication features that can connect to network 105 ande-mail server. For example, devices 130 and 150 can include a wearabledevice, a tablet computer, a wired phone, a mobile phone, a personal(e.g., laptop or desktop) computer, a streaming device, such as a gameconsole or other media streaming device, or the like. The server 110 hasone or more processors (not shown) that run code or software and/orexecute instructions to perform the functions of a meeting platform 112,and the presenter meeting device 150 and each of the participant meetingdevices 130 may run a local app or plugin for the meeting platform 112to allow content to be shared during a video conferencing session or anonline presentation.

The meetings platform (or meetings hosting module) 112 may include acollaboration service 114 or be able to call this service 114 during ahosted presentation or content sharing session/event to perform thefunctions described herein. The collaboration service 114, in turn, mayinclude or call an NLP module 118 on the server 110 (or accessible vianetwork 105), to process two or more questions to determine if theyshould be combined due to the relatedness of their content. The server110 is also shown to include data storage including a database 120(which may also be distributed in some cases on network 105). Thedatabase 120 is used to store a presentation recording 122 (e.g., a filecontaining the content presented during a meeting or trainingsession/event) as well as records 124 of data received from participants(e.g., a user ID, a question, a timing of a question, and a location ontheir display or its screen associated with the question). The database120 also stores generated collaboration records (or question reports)126 created by the collaboration service 114 for each received question(or for combinations of such questions if combined based on feedbackfrom the NLP module 118). During operations of the system 100, theserecords/reports 126 are transmitted by the collaboration service 114 tothe presenter meeting device 150 (or are accessible by this device) foruse in an operator of such a device 150 in responding to the userinput/questions.

As shown, each client device 130 includes a processor 132 executing codeor instructions (or running software) to provide a variety of functionsincluding a participant collaboration utility 140. The processor 132also manages operations of the device's I/O elements 134 that may beinclude a display 136 as well as input devices such as a touchscreen(which may be part of the display 136), a keyboard, a mouse, voicerecognition devices, and the lie to allow the user to provide inputincluding a content selection (e.g., touch or click the screen ofdisplay 136 during a presentation to indicate a portion of thepresentation) and user input (e.g., a question or remark or similarcontent related to the content selection). The display 136 is operatedduring the session/event hosted by platform 112 to display presentationcontent 139 (e.g., a slide, frames of a video, and so on) whileconcurrently speakers or audio output components in the I/O elements mayoutput corresponding audio content (e.g., speaking by an operator thepresenter meeting device 150).

The participant client device 130 also includes memory or data storage144 that has its access managed by the processor 132. During acollaboration session or event (e.g., a video conferencing class ormeeting), the participant collaboration utility 140 may functions tomonitor for any user interaction with the displayed content 139. Wheninteraction such as a user touching their touchscreen display 136 orclicking their mouse at a location in the screen/display 136 isdetected, the utility 140 acts to store a screen location 147 in memory144 associated with the selection (or user click). Also, the utility 140acts to determine and store an identifier 146 for the content 139displayed when the user selection or clicking occurred, and this may bea time of the selection.

The collaboration utility 140 may communicate this information (as shownas data 124 in database 120) to the server 110 for processing by thecollaboration service 114 for the service 114 to provide a new dialog orthe client-based utility 140 may respond by generating a new dialog, inplace of or over (or in a split window or the like) or new displayedcontent 139 that includes a screenshot of the content associated withthe identifier 146 (e.g., slide or frame shown in presentation streamwhen user selection/clicking occurred) along with a dialog box intowhich the operator of the participant client device 130 may enter theiruser input 148, which may take the form of a typed or spoken question,remark/comment, or the like related to the presentation contentidentified by the content identifier 146.

This new dialog may include a visual indicator (e.g., a dot, a circle,an “X”, or the like) of the screen location 147 captured by the utility140 from the user's input/selection via the I/O elements 134. The usermay accept this location and the resulting selection of a portion of thedisplayed content 139 of the presentation to associate with thecomment/input 148 or they may be allowed, in some embodiments, to movethe visual indicator on the screenshot so as to better associate theirquestion with the correct portion of the displayed content (e.g., movethe colored dot or circle to a different portion of a displayed slide toprovide more accurate context for their question).

The presenter meeting or client device 150 also includes a processor 152running code or software to provide a presenter collaboration utility152 that interacts with the collaboration service 114 to provide the newin-conference question summarizing features described herein. The device150 also includes I/O elements 156 includes input devices to allow apresenter to select and view differing questions and associated contextfrom the participant client devices 130. The I/O elements 154 include adisplay 156 upon which a generated GUI 138 is displayed that presents ordisplays content 139, which may include the presentation content of thecurrent session/event they are leading and which may include a questionreport list from the collaboration service providing links to questionreports/records (that provide question content along with an ID of theperson who submitted the question as well as context including theportion of the content associated with that question).

To this end, the memory 154 of the presenter client device 150 may storepresentation content 156 received from the collaboration service 114that is associated with a participant's question. In some cases, thiscontent 156 may be a screenshot associated with a question or it may bea plurality or set of slides, frames, or the like of the recordedpresentation 122. For example, a user may submit a question at 1 pmduring a presentation, and the received presentation content 156 mayinclude a predefined number of slides on either side of the slide thatwas shown at 1 pm or may include a video recording that includes theframe associated with 1 pm (e.g., provide a video recording of thepresentation from 12:58 pm to 1:02 pm or some other time window) so asto provide additional context related to the question to the presenterrather than a single screen shot. The received presentation content 156also may include audio in additional to visual content to assist thepresenter in understanding the specific context of the question or otheruser input during the presentation.

Additionally, as shown stored in memory 154, the collaboration service114 provides the presenter meeting device 150 with a collaborationrecord or question report 160 for each question (or combination ofquestion) from the participants (operators of devices 130), which may beorganized as a question report list in the displayed content 159 oftheir GUI 138. Each record 160 provides the operator of the device 150with a participant's input (e.g., question) 162 in an “as received” formor after some modifications by the NLP tool 118 to combine two or morequestions into a single one. The record 160 also includes a participantID 164 as well as a pointer to the content 170 of the presentation. Thepointer 170 may provide a timestamp 172 in the presentation to allow theutility 152 to retrieve the correct content in the received presentationcontent 156 (or this function may be provided by service 114) and ascreen location 174 to further identify the portion of the displayedcontent at a particular point in time of the presentation that isrelated to the user input/question 162. The displayed content 159 may bemodified to show a screenshot of the presentation content 156 associatedwith the timestamp 172 with a visual identifier of the screen location174 to assist the presenter in determining the portion of the contentassociated with the question/user input 162.

At this point in the description, it may be useful to explain operationsof a communications system (such as system 100 of FIG. 1) during acollaboration session or event (e.g., during a video conferencingsession to support a business meeting or online learning class or thelike). FIGS. 2A and 2B illustrate a display device 210 operating,respectively, to display collaboration session content (a slide during apresentation or the like) and to display a dialog to facilitate aparticipant entering a question.

As shown in FIG. 2A with screenshot 220, the display 210 is operated,during a collaboration session or presentation hosted by a meeting orvideo conferring platform (such as platform 112 of FIG. 1), to displaypresentation content 222 in a first part of the display 210 (or screenof the display 210). Concurrently, the display 210 may be operated toprovide in its GUI shown in screenshot 220 to provide boxes or windowsof supplemental content such as a first box 224 identifying a presenter(e.g., with their name or with their name and a still or video (live)image) and a second box 226 identifying the participant operating theclient device with the display 210 (again, with an identifier oridentifier and name). The presenter client device (not shown) would beoperated to provide shared content (e.g., content 222 in the first partof the display 210) and, in some cases, supplemental information asshown in boxes/windows 224, 226.

In a collaboration system (such as the system 100 of FIG. 1), thedisplay 210 is also operated to monitor for user input indicatingselection of a portion of the content in the first part of the screen ordisplay 210 that they wish to comment upon or to ask a questionregarding. As shown, the user/operator has operated their I/O devices(e.g., a mouse) to move a cursor 228 over a subset or portion 234 of thecontent in the first part of the display 210. The collaboration systemdetects when the user selects or clicks on this location of the cursor228 and, in response, modifies the GUI to indicate with a selectionlocation indicator 230 (here a dot or solid circle). The selection bythe user is an indication that they want to transmit user input to thepresenter of the content 222.

As shown in FIG. 2B with the screenshot 221, the collaboration systemoperates the display 210 to update the GUI to continue to show thepresent content of the collaboration session or presentation 222 and toalso generate and present a new dialog (or dialog box or window) 240. Inthis dialog 240, a thumbnail 242 of the presentation or session contentat the time the user made their selection is provided with the display210. In the thumbnail 242, the selection location indicator 230 isshown, and, from this, a leader line 245 is provided to a dialog box244. The use is able to enter user input (e.g., a question, a comment,or other input) 246 in this box 244 such as by typing, by use of voiceinput devices, and the like. When done entering the user inputtedcontent 246, the user may select in this GUI shown in screenshot 221 viabutton 250 to cancel the input or via button 252 to send the user input246 (along with their ID and the location of the cursor 228 at the timethe question/input was initiated) to the web server running thecollaboration service (service 114 of the meeting platform 112 in FIG.1). After the selection of the send button 252, the display 210 isoperated again in the operating mode of FIG. 2A (e.g., to display thepresent session/presentation content 222 and to remove the dialog 240).

FIG. 3 is a flow diagram of a method 300 of in-conference questionsummarization as may be performed by a communication or collaborationsystem of the present description such as the system 100 shown inFIG. 1. An initial process or step may include downloading collaborationutilities or plugins on each client device (presenter and participantdevices) to enable sharing of session or event content and to enable theclient-based and web or cloud-based collaboration service components towork together to monitor the session/event and to provide the questionsummarizing functions described herein.

The method 300 then includes at 310 the step of the participant or userof a session/event joining the session/event (or “meeting”), whichtypically will involve video conferencing. In step 310, their displaymay be operated to display content being presented or shared by thepresenter/facilitator of the meeting. In step 320, the user clicks onthe displayed presentation content, e.g., moves the cursor to a locationon the display's screen and selects (e.g., left clicks on their mouse)the content at that location to mark as being related to their userinput.

The method 300 continues with step 330 involving the display device ofthat user being operated to display or present (such as by operations ofthe client-based collaboration utility alone or in combination with theweb-based collaboration service) a screenshot of the presentationincluding where (the location) they clicked. The coordinates on thescreen of the click/selection of content are stored at least temporarilyon the client device and later in web server in a question reportgenerated for this user's input or question. In step 340, the method 300continues with the user typing in (or otherwise entering) their question(or user input) such via a presented dialog box in the GUI provided inthe display device.

The question report is generated, by the client-side utility or by theweb-based collaboration service. The question report or collaborationrecord may include a timestamp (or other identifier of the user-selectedcontent in the presentation), a screenshot with a click point,coordinates of the click, the user's ID, and the question asked by theuser. The question report is then transmitted over the communicationsnetwork to the server (e.g., one hosting the meeting/session/event andrunning the collaboration service or one running the collaborationservice in a distributed manner).

In step 370, the question report is analyzed to determine if it is aduplicate or is cumulative of another already received question fromanother user. This may involve comparing the content associated with thequestion and/or the timing of the question. In other words, twoquestions may be considered potentially related and requiring furtheranalysis if they both fall within a predefined time window within thepresentation/event, e.g., two timestamps within 5 minutes of each other,within 3 minutes of each other, within 30 seconds of each other, or thelike. Typically, the content of the presentation associated with theclick/selection location should be matching, too, prior to furtheranalysis for a duplicative or cumulative question. The analysis may, iftiming and content are determined to be “matching,” involve comparingthe content of the user input/question to determine whether an adequatecorrelation is found to combine the two questions or if they should bekept as separate records, and such analysis may be involve NLPprocessing (e.g., with NLP module 118 of system 100 of FIG. 1 and asexplained in more detail below).

If not a duplication at 370, the question report is retained in theserver database at 360, and the method 300 continues at 380 with thequestion report being sent to (or updated on) the presenter's clientdevice as a question report list showing all question reports receivedso far during the ongoing presentation (or at the end of thepresentation in some cases). The question report listing may include acount of how many have asked this question when duplicates areidentified in step 370 and may include the user IDs of those submittingthe questions. In step 390, the presenter may have their GUI in theirdisplay device updated so they can view the question report list alongwith or separately from the presentation content they are currentlysharing/presenting. The presenter in step 390 may select a question toanswer and be provided the context of the question such as a screenshotof the content corresponding with the timestamp/user's content selectionand a click location as well as the question/user input content.

FIG. 4 is another flow diagram of a method 400 of in-conference questionsummarization as may be performed by a communication or collaborationsystem of the present description similar to that shown in FIG. 3 butshowing the user of NLP subprocessing at 444. FIG. 5 is a flow diagramshowing an exemplary implementation of the NLP subprocessing 444 thatmay be used in the methods of FIGS. 3 and 4 and as may be carried out bythe operation of the system 100 of FIG. 1.

In the method 400, the client-side collaboration utility acts to collectuser input (or a “question”) 410, and this input 410 may include avariety of user-provided content including comments 412, questions 414,notes 416, and chat 418. The utility processes this as part of an intakeprocess/step 420 for the question report that is transmitted to theserver 430 hosting the presentation/session/event (e.g., an onlinecontent sharing with video conferencing). The intake 420 includescapturing a timestamp 426 defining when in the presentation the userindicated (e.g., by clicking on their screen showing the presentationcontent) they wished to provide user input. The server 430 may alsostore, as part of the question report, the user ID for theparticipant/user providing the input 410, a screenshot of thepresentation at the time in the presentation corresponding with thetimestamp 426, cursor screen coordinates, and the user input 410 (orportions thereof).

The question report 420 is then provided by the server 430 to an NLPsubprocess 444 for analysis/processing to determine whether thequestion/user input is unique or whether it is duplicative or cumulativeto one or more previously received user inputs in question reportsstored by the server 430 during the presentation/session/event. Theprocessed question report is then output at 450 for delivery as part ofa question report list to the client device of the presenter, e.g., withduplicates deleted or combined with others, a count increase of numberof times the question/content has been presented, and a listing of userIDs associated with the matching content.

In the method 500, the server 430 feeds the question report, and itscomponents, to the NLP subprocess 444 for pre-processing at step 510.The pre-processing 510 may involve text tokenization, normalization,noise removal, and lemmatization, and/or involve NLP pre-processing asdescribed athttps://nlp.stanford.edu/IR-book/html/htmledition/tokenization-1.html orthe like. The NLP subprocess 444 then proceeds at 520 with cleaning ofthe content to be processed (of the user input/question). The cleaning520 removes stop words (e.g., $, a, the, an, this, and so on), specialcharacters, numbers, and the like. There may be exceptions when thequeries or comments are idiomatic or terse, and, in these cases, thecleaning 520 may retain these as they may be useful for providing fullcontext to understanding the user inputted content/question.

The NLP subprocess 444 continues at 530 with applying a trained deep orshallow learning model to the content of the user input. This NLPanalysis is understood in the art with exemplary implementations anddescription provided, for example but not as a limitation, athttps://developer.ibm.com/technologies/artificial-intelligence/articles/cc-machine-learning-deep-learning-architectures/.Next, at 540, the NLP subprocess 444 generates higher levelrepresentations using the output from steps 510, 520, and 530. Anexample of such NLP processing can be found athttp://stefanfrank.info/pubs/Schmalhofer.pdf, and with regard to thestep 540: (a) cleaned and pre-processed text is fed into deep neuralnetworks of variable depth; (b) by variable depth, each neural layerabstracts a different meaning from the analyzed input; (c) generally,the method 444 leverages activation outputs because it provides higherlevel abstraction of the varied inputs; (d) the higher level abstractionmakes the representations of linguistic/semantic elements simpler; (e)these vector representations can be shallow (Word2Ved, Glove, and thelike), contextual (e.g., USE), or averaged (deep averaging network orthe like) depending on the use case; and (f) unsupervised clusteringalgorithms can then be used to divide these representations (questions)into logical groupings.

With the output of step 540 generated, the question summarizing method500 continues at 550 with a check of the timestamp and cursorcoordinates with other previously processes question reports (userinputs/questions). The closeness of timestamps (e.g., two (or more)questions entered during a presentation within a predefined time window)and/or coordinates (another set of coordinates within a predefinedradius of the present question's coordinates) allows for better groupingof similar questions. If the check at 550 indicates a possibleduplication (e.g., content being “questioned” is matching or nearly so),the method 500 continues at 560 with scoring of the semantic similaritybetween the presently being processed user input question and apreviously received and processed user input question (with matchingtimestamp and/or cursor coordinates).

If the scoring indicates a likely duplication or cumulation of contentbetween the two questions (as may be performed as described, forexample, at https://dbis.ipd.kit.edu/download/2017.3.pdf) the method 500continues at 570 with grouping of the similar question reports.Questions may be grouped according to mutual similarity/dissimilarity ofword/sentence level representations of the questions. The groupings canvary according to the type of similarity scoring used in step 560 (e.g.,Euclidean distance, L2 norm, Jaccard, and the like). In step 580, thedatabase managed by the collaboration service server is updated toreflect the grouping 570 (or decision in method 500 not to group as maybe made at 550 if the check fails or at 560 if scoring fails to indicatean adequate similarity). The method 500 continues at 590 with generationof a response/output 450, with a response engine in the NLP module,which may include neural text generation or a statistical method (phraselibrary used in different contexts) and/or may be carried out asdiscussed at https://cs.stanford.edu/˜zxie/textgen.pdf.

As will be understood by those skilled in the arts, many companies use avideo conferencing or meeting tool when broadcasting a presentation to alarge group of people. Typical use cases would be audio or videoconferencing where content is presented and shared with othersincluding, but not limited to, townhall presentations, business updates,or product demonstrations. During the presentation, a member of thegroup may have questions about a specific portion of the presentation.With tools prior to the present description, the user could raise theirhand during the presentation or type in a question into the system. Thepresenter would then be shown a list of all the questions that wereasked. Depending on how long the presentation was, the presenter may nothave a context of what the question is related to in the presentationcontent. If users are only allowed to ask their questions at the end ofthe presentation, they may forget their question because of the lengthof the presentation.

The present method and system provides a way to allow audience membersto ask questions while also providing a context for the questions to thepresenter. At any time during the presentation, the user can choose toclick anywhere on their screen. The client application will take ascreenshot of the current state of the presentation being shared,whether this is a video, slide, demonstration, or other shared contentdisplayed on the screen of their client device. The client applicationwill also record the placement of the user's click (cursor location) onthe presentation screen. The timestamp of when this occurred is alsorecorded. A new dialog will then open that has a preview of thescreenshot depicting the click point using, for example, a coloredcircle whose location was determined using the placement of the user'scursor (via a mouse, touchscreen, or the like) at the time at which theyclicked/selected the input to mark for a question/user input. There mayalso be a dialog box allowing the user to ask/enter a question. Once theuser has typed (or otherwise entered) their question and submitted it, aquestion report is sent that includes a user ID, a timestamp of when thescreenshot was taken, the screenshot with the user' click point, and thequestion itself.

When a presentation/meeting starts, the server hosting the meeting (oranother providing the collaboration service) will automatically createan empty question report list against the meeting (or linked to thesession/event) typically using the meeting ID to link the report list tothe meeting. The server is then ready to accept incoming questionreports from participants' client devices. Once a user/participant hassent their question report, it is received by the server. If thequestion report list is empty, this question report is automaticallyadded to the empty list.

Otherwise, in most embodiments, the question report will be compared(such as via NLP analysis) against all the exiting question reports inthe list to see if there has already been a similar question asked byanother user. For example, the timestamp will be analyzed to determinewhere the question fits chronologically in the received questions and ifthere are other questions asked around the same timestamp (e.g., withina predefined time window). If a question is determined to be in the sametimeframe or window, the screenshot and click location coordinates willbe compared to one another to determine if they too are similar. If so,then the question/user input content will be analyzed using NLP or othertechniques to determine if there are shared key words and/or topics. Ifthe question is found to include similar content or to be a duplicate(or at least cumulative), the question is associated to the matchedquestion report and marked as duplicate. If the question is notduplicative, it is inserted into the ordered question report list at theappropriate spot based on its timestamp (list questions chronologicallyin many applications).

In the presenter's view of the presentation on their client device, theymay be allowed via their GUI to operate a toggle button (or other inputelement) to allow the presenter to show or hide questions that have beenasked during (or after) their presentation. The question report list isupdated on an ongoing basis to be current (or nearly real-time) duringthe presentation as the server receives more questions from other users.The presenter can view a question that has been asked by clicking on ashow button. If the presenter would prefer to only see questions duringa post-presentation Q&A period, they can choose in their GUI to keep thequestion report list hidden and view them at their leisure such as afterthey complete sharing of content.

The list that the presenter is shown is ordered by the timestamps andmay include a thumbnail of the captured screenshot and/or thequestion/user input content. If the presenter clicks on any question inthe list, a new dialog box will, in some embodiments, be shown with thefull screen shot, question, and the user's name (which may be determinedby the system using the user's ID) for the participant who asked thequestion/provided the user input. If there are duplicate questionsassociated with the question report chosen, a button may be provided inthe presenter's GUI to allow them to view the duplicate questions. Whenviewing a duplicate, the presenter may be enabled via their GUIconfiguration to choose to remove the duplicate tag if they find thequestions to be different and/or to be directed toward differingcontent. Once the presenter has answered the chosen question, their GUImay be further configured to allow them to click or select an answeredbutton to specify that they have already answered this question, and thequestion report list may be updated to indicate which questions havebeen answered or at least discussed or addressed by the presenter.

As used herein, the terms application, module, analyzer, engine, and thelike can refer to computer program instructions, encoded on computerstorage medium for execution by, or to control the operation of, dataprocessing apparatus. Alternatively or additionally, the programinstructions can be encoded on an artificially-generated propagatedsignal, e.g., a machine-generated electrical, optical, orelectromagnetic signal, which is generated to encode information fortransmission to suitable receiver apparatus for execution by a dataprocessing apparatus. A computer storage medium can be, or be includedin, a computer-readable storage device, a computer-readable storagesubstrate, a random or serial access memory array or device, or acombination of one or more of the substrates and devices. Moreover,while a computer storage medium is not a propagated signal, a computerstorage medium can be a source or destination of computer programinstructions encoded in an artificially-generated propagated signal. Thecomputer storage medium can also be, or be included in, one or moreseparate physical components or media (e.g., solid-state memory thatforms part of a device, disks, or other storage devices).

The present invention has been described above with reference to anumber of exemplary embodiments and examples. It should be appreciatedthat the particular embodiments shown and described herein areillustrative of the invention and its best mode and are not intended tolimit in any way the scope of the invention as set forth in the claims.The features of the various embodiments may stand alone or be combinedin any combination. Further, unless otherwise noted, various illustratedsteps of a method can be performed sequentially or at the same time, andnot necessarily be performed in the order illustrated. It will berecognized that changes and modifications may be made to the exemplaryembodiments without departing from the scope of the present invention.These and other changes or modifications are intended to be includedwithin the scope of the present invention, as expressed in the followingclaims.

We claim:
 1. A method of summarizing questions during an onlineconference comprising: over a communications network, hosting acollaboration session in which digital presentation content is shared bya presenter client device with participant client devicescommunicatively coupled to the communications network; with a serverlinked to the communications network, receiving a question report fromone of the participant client devices, wherein the question reportincludes identification data of a portion of the presentation contentand user input content associated with the portion of the presentationcontent; and with the server, transmitting the question report to thepresenter client device for access by an operator of the presenterclient device during the collaboration session.
 2. The method of claim1, wherein the identification data includes a timestamp and a screenshotof the presentation content displayed on a display of the one of theparticipant client devices at a time matching the timestamp.
 3. Themethod of claim 2, wherein the identification data further includescursor coordinates on the display corresponding to a user interactionwith the presentation content of the screenshot at the time matching thetimestamp.
 4. The method of claim 3, further including modifying thescreenshot to include a visual marker at a location based on thecoordinates.
 5. The method of claim 1, further comprising comparing thereceived question report with a previously received question report todetermine whether the received question report is duplicative of thepreviously received question report and when duplicative combining thereceived question report with the previously received question reportprior to the transmitting step.
 6. The method of claim 5, wherein thecomparing step comprises performing natural language processing (NLP) onthe user input content to determine whether user input content in thepreviously received question report is a match.
 7. The method of claim5, wherein the comparing step comprises determining whether thetimestamp is within a predefined time window containing a timestamp ofthe previously received question report.
 8. The method of claim 5,wherein the identification data further includes cursor coordinates onthe display corresponding to a user interaction with the presentationcontent of the screenshot at the time matching the timestamp and whereinthe comparing step comprises determining whether the coordinates arewithin a predefined distance from cursor coordinates in the previouslyreceived question report.
 9. The method of claim 1, further comprising,prior to the receiving step, displaying the presentation content adisplay screen of the one of the participant client devices, respondingto a user interaction with the display screen during the displaying bycollecting a timestamp and a cursor coordinates associated with the userinteraction and further by displaying a screenshot of the presentationcontent displayed at a time matching the timestamp along with a dialogbox, receiving the user input content via the dialog box, andtransmitting the question report to the server including the timestamp,the cursor coordinates, the screenshot, and the user input content. 10.The method of claim 1, wherein the user input content includes at leastone of a question, a comment, a note, and chat.
 11. An electroniccommunication system for summarizing questions during an onlineconference comprising: a web server including a meetings platformhosting an online communications session on a digital communicationsnetwork; a presenter client device sharing presentation content in theonline communications session; a participant client devicecommunicatively coupled to the digital communications network receivingthe presentation content and displaying the presentation content upon adisplay, wherein the participant client device runs a collaborationutility monitoring for a user interaction with the display during thedisplaying of the presentation content and, in response, to display ascreenshot of the presentation content of the presentation content and adialog box for receiving user input content; and a collaboration serviceon the web server receiving a question report from the participantclient device comprising the user input content and, in response,transmitting the question report to the presenter client device fordisplay in a question report list.
 12. The electronic communicationsystem of claim 11, wherein the collaboration utility on the participantclient device further captures a timestamp when the user interactionwith the display is detected and wherein the question report transmittedto the presenter client device comprises at least one of the timestampand a screenshot of the presentation content displayed on theparticipant client device at time of capture of the timestamp.
 13. Theelectronic communication system of claim 11, wherein the collaborationutility on the participant client device further captures cursorcoordinates associated with the user interaction with the display,wherein the screenshot includes a visual marker displayed at the cursorcoordinates, and wherein the question report transmitted to thepresenter client device includes the screenshot with the visual marker.14. The electronic communication system of claim 11, wherein thecollaboration service compares the received question report with apreviously received question report to determine whether the receivedquestion report is duplicative of the previously received questionreport and when duplicative combining the received question report withthe previously received question report prior to the transmitting of thequestion report to the presenter client device.
 15. The electroniccommunication system of claim 14, wherein the comparing comprisesperforming natural language processing (NLP) on the user input contentto determine whether user input content in the previously receivedquestion report is a match.
 16. The electronic communication system ofclaim 15, wherein the comparing further comprises at least one ofdetermining whether a timestamp provided in the received question reportis within a predefined time window containing a timestamp of thepreviously received question report and, when the received questionreport includes cursor coordinates on the display corresponding to theuser interaction, determining whether the coordinates are within apredefined distance from cursor coordinates in the previously receivedquestion report.
 17. The electronic communication system of claim 11,wherein the question list report comprises a plurality of questionreports from the participant client device and additional participantclient devices participating in the online communications session,wherein the plurality of question reports are arranged chronologicallyby timestamps indicating when the participant client devices wereoperated to select a portion of the presentation content to commentupon, and wherein the presenter client device is operable to display oneof the question reports including an identifier of the portion of thepresentation content and the user input content.
 18. A method ofsummarizing questions during an online conference comprising: with aserver linked to a communications network, receiving a question reportfrom a participant client device participating in a web-basedcollaboration session involving sharing of presentation content, whereinthe question report includes identification data of a portion of thepresentation content and user input content associated with the portionof the presentation content; comparing the received question report witha previously received question report to determine whether the receivedquestion report is duplicative of the previously received questionreport and, when duplicative, combining the received question reportwith the previously received question report prior to the transmittingstep to provide a combined question report; and with the server,transmitting the received question report or the combined questionreport to a presenter client device sharing the presentation content foraccess during the collaboration session.
 19. The method of claim 18,wherein the comparing step comprises performing natural languageprocessing (NLP) on the user input content to determine whether userinput content in the previously received question report is a match. 20.The method of claim 18, wherein the identification data includes atimestamp and a screenshot of the presentation content displayed on adisplay of the one of the participant client devices at a time matchingthe timestamp, wherein the identification data further includes cursorcoordinates on the display corresponding to a user interaction with thepresentation content of the screenshot at the time matching thetimestamp, and further including modifying the screenshot to include avisual marker at a location based on the coordinates.
 21. The method ofclaim 20, wherein the comparing step comprises at least one ofdetermining whether the timestamp is within a predefined time windowcontaining a timestamp of the previously received question report anddetermining whether the coordinates are within a predefined distancefrom cursor coordinates in the previously received question report. 22.The method of claim 18, further comprising, prior to the receiving step,displaying the presentation content a display screen of the participantclient device, responding to a user interaction with the display screenduring the displaying by collecting a timestamp and a cursor coordinatesassociated with the user interaction and further by displaying ascreenshot of the presentation content displayed at a time matching thetimestamp along with a dialog box, receiving the user input content viathe dialog box, and transmitting the question report to the serverincluding the timestamp, the cursor coordinates, the screenshot, and theuser input content.